The present invention relates to a data coding method and apparatus capable of coding video and audio data for special reproduction, to a data decoding method and apparatus for reading recorded video and audio data from an optical disk, a magnetic disk or the like and reproducing the read data in a special mode, and further relates to a recording medium where coded data is recorded in a manner so as to be reproducible in a special mode.
Digital picture signals or the like to be recorded on a disk in a digital video disk (hereinafter referred to as DVD) system, are compressed and coded using the MPEG (Motion Picture coding Experts Group) method.
FIG. 14A is a schematic representation of the structure of inter-frame prediction structure used in the MPEG system. In this example, one GOP (Group of Pictures) is composed of, e.g., fifteen frames, which include one frame of an I-picture (intraframe encoded picture), four frames of P-pictures (forward interframe prediction encoded pictures), and the remaining ten frames consisting of B-pictures (forwarded and backward, bidirectional prediction encoded pictures).
As used herein, the I-picture is an intra-picture coded picture in which either one frame or one field is data compressed using intra-frame or intra-field coding consistent with the MPEG system; the P-picture is an inter-picture forward predictive coded picture in which either a frame or a field is compression encoded using inter-frame or inter-field coding with reference to the temporally preceding frame or field (I-picture or P-picture) already coded; and the B-picture is a bidirectionally predictive coded picture in which either a frame or a field is compression encoded using inter-frame or inter-field coding with reference to the temporally preceding and succeeding frames or fields.
More specifically, as indicated by arrows in the diagram, an I-picture I.sub.0 is coded by intra-frame processed by itself and without references to any other frame; a P-picture P.sub.0 is coded by inter-frame prediction with reference to the I-picture I.sub.0 ; and a P-picture P.sub.1 is coded by inter-frame prediction with reference to the P-picture P.sub.0. Further, B-pictures B.sub.0, and B.sub.1 are coded by inter-frame prediction with reference to both the I-picture I.sub.0 and the P-picture P.sub.0 ; and B-pictures B.sub.2 and B.sub.3 are coded by inter-frame prediction with reference to both the P-picture P.sub.0 and the P-picture P.sub.1. Similarly, subsequent pictures are coded by such prediction in the manner indicated by arrows.
In decoding the predictive-coded pictures mentioned, the I-picture is decoded alone since it is not coded with reference to any other frame. However, a preceding I-picture or a preceding P-picture is required to decode a given P-picture because a P-picture is predictive-coded with reference to the temporally preceding I-picture or P-picture. Similarly, preceding and succeeding I-pictures or P-pictures are required to decode a given B-picture because a B-picture is coded with reference to the temporally preceding and succeeding I-pictures or P-pictures.
For this reason, to provide proper decoding, positions of the pictures on a record medium are changed from FIG. 14A to the positions illustrated in FIG. 14B so that the pictures required for decoding are decoded in advance.
As illustrated in the diagram, such positional changes are made so that the I-picture I.sub.0 precedes the B-pictures B.sub.-1 and B.sub.-2 since the I-picture I.sub.0 is required to decode the B-pictures B.sub.-1 and B.sub.-2 and also the position of the P-picture P.sub.0 is changed to precede the B-pictures B.sub.0 and B.sub.1 since the decoding of B-pictures B.sub.0 and B.sub.1 requires the I-picture I.sub.0 and the P-picture P.sub.0. Similarly, other pictures are positionally changed so that the P-picture P.sub.1 precedes the B-pictures B.sub.2 and B.sub.3 since the P-pictures P.sub.0 and P.sub.1 are required to decode the B-pictures B.sub.2 and B.sub.3, and also the P-picture P.sub.2 precedes the B-pictures B.sub.4 and B.sub.5 since the decoding of the B-pictures B.sub.4 and B.sub.5 requires the P-pictures P.sub.1 and P.sub.2. In the same manner, positional changes are made so that the P-picture P.sub.3 precedes the B-pictures B.sub.6 and B.sub.7.
The video data composed of the I-picture, P-pictures and B-pictures arranged in the order of FIG. 14B, and other data including audio data and subtitle (captions) data, are packetized (multiplexed) and recorded on a recording medium such as a disk or are transmitted on a transmission channel. The code quantity of each frame in the picture data is not fixed among pictures and depends on the complexity or flatness of the individual picture. Typically, an I-picture is represented by more data than a P-picture which is represented by more data than a B-picture.
FIGS. 15A to 15C show one example of how the data is packetized. In these diagrams, FIG. 15A represents an MPEG2 system stream which is multiplexed after packetization; FIG. 15B represents the content of a video packet in the multiplexed stream; and FIG. 15C represents an MPEG2 video stream of a video layer.
In each of picture data V, V+1, V+2, . . . and so forth constituting the video layer of FIG. 15C, picture header information and picture coding extension information are affixed at the leading position. In the example shown, a video stream ranging from the position identified as D1 to the position identified as D3 of the video layer forms one video packet with a packet header affixed at its leading position, and a video stream ranging from the position D3 to the position identified as D5 of the video layer forms another video packet with a packet header affixed at its leading position.
Video packets thus packetized are multiplexed with audio packets and subtitle packets to thereby form the MPEG2 system stream shown in FIG. 15A.
FIG. 16 shows the contents of a picture header, and FIG. 17 shows the contents of a picture coding extension.
In the picture header, there are items of information such as a unique picture.sub.-- start.sub.-- code, temporal.sub.-- reference (TR) (which is a time-series serial number given per picture), and picture.sub.-- coding.sub.-- type (I-, P- or B-picture).
In the picture coding extension, there are items of information such as a unique extension.sub.-- start.sub.-- code, a unique extension.sub.-- start.sub.-- code.sub.-- identifier, picture.sub.-- structure, top.sub.-- field.sub.-- first, progressive.sub.-- frame, etc.
As for the picture data, two data structures may coexist: a frame structure where one picture is composed of one frame and a field structure where one picture is composed of two fields. Whether the picture data has a frame structure of one frame per picture or a field structure of two fields per picture can be identified from the following three items of information, i.e., (1) presence of GOP header, (2) temporal.sub.-- reference (TR) in picture header, and (3) picture.sub.-- structure in picture coding extension.
FIG. 18 is a block diagram illustrating one example of data decoding apparatus adapted to perform special reproduction of data, such as slow picture playback, fast playback, reverse playback and the like. An optical disk 1 is rotatable by a spindle motor (not shown) at a predetermined rotation rate, and a laser beam is projected from a pickup 2 to a track on the optical disk 1, so that the MPEG compressed digital data recorded on the track is read therefrom. The digital data is processed by a demodulator circuit 3 which demodulates eight to fourteen modulation (EFM) and supplied to a sector detection circuit 4. The output of the pickup 2 also is supplied to a phase-locked loop (PLL) circuit 9, where a clock signal is reproduced and supplied to the demodulator circuit 3 and to the sector detection circuit 4.
The digital data recorded on the disk 1 includes multiplexed streams recorded in units of a fixed-length sector, with a sector sync and a sector header affixed to the beginning of each sector. The sector detection circuit 4 detects each of the sectors from the sector sync and the sector address from the sector header; and this information is supplied to a control circuit 6.
The demodulated digital data is supplied via the sector detection circuit 4 to an ECC (error correction) circuit 33 which executes error detection and correction. The ECC circuit 33 supplies error-corrected data to a ring buffer 5 to be written therein under control of the control-circuit 6.
The output of the ECC circuit 33 also is supplied to a stream detector 50, which determines the picture type from the data stream picture header read from the disk 1 in a special reproduction mode and then supplies picture type information to the control circuit 6. In response to this information, the control circuit 6 executes its control operation in such a manner that, in the special reproduction mode, the data of the I-picture and the data of the succeeding two P-pictures are written in the ring buffer 5.
A focus control circuit (not shown) and a tracking servo circuit 8 control focusing and tracking of the pickup 2, respectively, under the control of a system controller (not shown) in response to a focus error signal and a tracking error signal obtained from the information read by the pickup 2.
In accordance with the sector address of each sector detected by the sector detection circuit 4, the control circuit 6 designates, by a write pointer WP, a write address for writing the corresponding sector in the ring buffer 5. Moreover, in accordance with a code request signal obtained from a video code buffer 10 (FIG. 18B), the control circuit 6 further designates, by a read pointer RP, a read address of the data written in the ring buffer 5. The control circuit 6 is adapted to read the data from the position of the read pointer RP and supplies the read data to a demultiplexer 32.
Since the coded data recorded on the disk 1 comprises multiplexed video, audio and subtitle data, the demultiplexer 32 separates the data read thereto from ring buffer 5 into the video data, the audio data and the subtitle data, and then supplies the respective data to a video decoder 20 (FIG. 18B), an audio decoder (not shown), and a subtitle decoder (not shown). The video decoder 20 stores the video data in the video code buffer 10.
Thereafter, the data stored in the video code buffer 10 is supplied to a picture header detector 34 which detects the picture header. The detected picture header information is further used to identify the picture type (I, P or B picture) of the video data and the temporal reference (TR) which signifies the frame order in the GOP. A picture data selection circuit 35 selects only the I-picture and the P-picture as identified by the picture type information supplied from the picture detector 34 in the special reproduction mode, and supplies the selected picture data to an inverse VLC (variable length coding) circuit 11. In a normal reproduction mode, the picture data selection circuit 35 is controlled to deliver all of the picture data to inverse VLC circuit 11 without any pre-selection.
The data supplied to the inverse VLC circuit 11 is processed using inverse VLC; and then is supplied to a dequantizer 12. Code request signals are returned to the video code buffer 10 from the inverse VLC circuit to permit new data to be transferred from the video code buffer 10.
Further, the inverse VLC circuit 11 outputs a quantization step size to the dequantizer 12 and outputs motion vector information to a motion compensator 15. The quantization step size and motion vector information are included with the video data. The dequantizer 12 dequantizes the input data in accordance with the designated quantization step size and outputs the dequantized data to an inverse DCT (discrete cosine transform) circuit 13. The inverse DCT circuit 13 processes the dequantized data using inverse DCT to recover video information, and supplies the recovered video information to an adder 14.
The adder 14 adds the output of the inverse DCT circuit 13 and the output of the motion compensator 15 in accordance with the picture type (I, P or B) and supplies the result, i.e., motion-compensated video data, to a frame memory bank 16.
Thereafter, the data read from the frame memory bank 16 is rearranged in the original frame order (as shown in FIG. 14A by switch 16E). The rearranged data is supplied to a digital-to-analog (D/A) converter 17 which converts the data into an analog video signal to be displayed on a display device 18.
Returning to FIG. 18A, the output of the ECC circuit 33 is supplied to a stream detector 50 which detects the picture type from the stream data read from the disk 1 and supplies the picture type information to the control circuit 6. In response to this information, the control circuit 6, in the special reproduction mode, writes into the ring buffer 5, the data of the I-picture and the succeeding two P-pictures.
Accordingly, three frames corresponding to the I- and two P-pictures at the beginning of each GOP are written at high speed into the ring buffer 5, and this data can be acquired and decoded by the decoder 20 at any desired timing, thereby enabling efficient decoding of data in the special reproduction mode.
For example, suppose that a reverse reproduction is started with the P-picture P.sub.3 of the original frame order shown in FIG. 14A. It is necessary to display the decoded pictures in the following order:
P.sub.3 .fwdarw.B.sub.7 .fwdarw.B.sub.6 .fwdarw.P.sub.2 .fwdarw.B.sub.5 .fwdarw.B.sub.4 .fwdarw.P.sub.1 .fwdarw.B.sub.3 .fwdarw.B.sub.2 .fwdarw.P.sub.0 .fwdarw.B.sub.1 B.sub.0 .fwdarw.I.sub.0 .fwdarw.. . .
However, since each P-picture is coded by inter-picture prediction as described, the pictures I.sub.0, P.sub.0, P.sub.1 and P.sub.2 need to be decoded before decoding the P-picture P.sub.3. Similarly, the P-pictures P.sub.2 and P.sub.3 need to be decoded prior to decoding the B-picture B.sub.7. Therefore, if reverse reproduction is to be performed by decoding each picture merely once, as in normal reproduction, it becomes necessary to employ a frame memory bank 16 of great capacity which is capable of storing as many frames as there are pictures constituting a GOP.
The storage capacity of the frame memory bank 16 must be increased beyond what is required in a normal reproduction mode to meet such requirement. Further, the decoded data must be sequentially stored in the frame memory bank to deliver the pictures in the proper order of reverse reproduction.
Although other techniques of reverse reproduction may be adopted to perform merely with the I- and P-pictures thus skipping the B-pictures, the necessity of storing more frames than what is required for normal reproduction still exists.
For this reason, the data decoding apparatus of FIG. 18 operates to perform reverse reproduction using the same frame memory bank as utilized in normal reproduction, i.e., using three memory elements in the example of FIG. 18 to store one I-picture and two temporally succeeding P-pictures. The stream detector 50 provided for this purpose writes the I-picture and the two succeeding P-pictures in the ring buffer 5. However, this makes the construction and operation of the steam detector 50 for detecting the I-picture and the two succeeding P-pictures more complicated.
FIG. 15A shows the packetized (multiplexed) MPEG2 system stream. When a packet of the MPEG2 video stream is defined at a position D3 as shown in FIG. 15C during the packetization process, the picture header and the picture coding extension of the picture data (V+2) are spread over two packets, as shown in FIG. 15B.
If the picture header and the picture coding extension are spread over two video packets, it becomes necessary to detect two video packets to obtain the requisite items of information of the picture. Further, and as shown in FIG. 15A, another packet (e.g., audio packet) may exist between the two video packets to complicate the detection process, thereby complicating the construction and operation of the stream detector 50.
According to the MPEG2 technique, video data in a frame structure where one picture is composed of one frame, and video data in a field structure, where one picture is composed of two fields, can be intermingled. Since a picture header is affixed to every field, the picture headers and the picture coding extensions of two consecutive pictures must be read to determine the data structure of the video data.
Therefore, a determination is made to ascertain whether the picture data is formed in the frame structure or the field structure on the basis of the aforementioned three items of information, i.e., (1) presence of GOP header; (2) temporal reference (TR) in the picture header; and (3) picture.sub.-- structure information in picture coding extension.
A detailed explanation will be given below with regard to a method of differentiating between a frame structure and a field structure.
FIGS. 19A and 19B show video data in the field structure and frame structure formats, respectively. In the field structure format, one frame of video data is composed of two fields of picture data, to each of which are affixed a picture header and a picture coding extension. In the frame structure format, one frame of video data is composed of one frame of picture data to which a picture header and a picture coding extension are affixed.
In the field structure format, the numerical values of the TR information in the respective picture headers of the picture data pair are set equal to each other. The picture.sub.-- structure information in the picture coding extension is "01" and "10" for Top Field and Bottom Field, respectively, as shown in FIG. 20. Further, the picture.sub.-- structure information in the picture coding extension of the frame structure is "11" as shown in FIG. 20.
The format (field or frame structure) of the picture data may be ascertained by first reading the GOP header at the GOP start position and then reading the picture.sub.-- structure information of the picture coding extension at the beginning of that picture data.
Although the picture data in the frame structure can be loaded into the ring buffer 5 (FIG. 18A) by detecting a single frame, it is difficult to similarly load the video data in the field structure, where a pair of picture data constitutes one frame of video data, because the paired picture data must be detected before it can be properly loaded. Consequently, the TR information in each picture header is read to find two picture data units with numerically equal values of TR. When such a pair is found, they are identified as paired picture data and then are loaded.
The paired field-structure picture headers are arranged in either of two different orders: top/bottom and bottom/top. Such arrangements will now be described with reference to FIG. 21. A GOP header (GOP H), an I-picture of frame structure, a B-picture of field structure, another B-picture of field structure, a GOP header spaced apart therefrom, and an I-picture of field structure, a GOP header, another I-picture of field structure . . . and so forth are sequentially recorded.
For example, where a total of three frames (one I-picture and two succeeding P-pictures) are loaded into ring buffer 5 (FIG. 18A), the I-picture in frame structure next to the top (first) GOP header is detected and identified from the GOP header, the picture.sub.-- coding.sub.-- type in the picture header, and the picture.sub.-- structure information ("11" in the case of frame structure) in the picture coding extension at the beginning of the picture data.
When access is made for playback at the position identified as random access 1 in the bit stream, the picture header and the picture coding extension of the first field-structure B-picture are read out. At this time, the TR expressed as "0" is also read out. Subsequently the picture header and the picture coding extension of the second field-structure B-picture are read out as well as the TR expressed as "0". Since the TR value of the two field-structure B-pictures are equal, they are detected as paired data.
When access is made at the position identified as random access 2 in the bit stream, the picture header and the picture coding extension of the first picture are read along with the TR expressed as "0". Subsequently the picture header and the picture coding extension of the next picture are read out along with the TR expressed as "1". Since the respective numerical values of the TR are not coincident with each other, the data of the two field-structure pictures are not detected as paired data.
If access is made at the position identified as random access 3 in the bit stream, the respective numerical values of the TR in the two picture headers are coincident (TR=1) with each other as in the foregoing case of random access 1, thus the pictures are detected as a pair. If a picture structure in the picture coding extension of "01" or "10" is detected, it is regarded as a field structure, and a paired data is detected.
If access is made at the position identified as random access 4 in the bit stream, the picture header and the picture coding extension of the first picture are read out along with the TR expressed as "0". This picture data is regarded as a field-structure I-picture in accordance with the picture coding type information in the picture header and the picture.sub.-- structure information in the picture coding extension.
After subsequent detection of the GOP header, the picture header and the picture coding extension of the next picture are read out along with the TR expressed as "0". Here, the respective numerical values of the TR of the two consecutive pictures are coincident with each other, but these two pictures are not considered as paired because there exists a GOP header between the two pictures. It should be noted that the TR is reset to "0" if a GOP header exists and no GOP header is interposed between a paired picture.
As described, the stream detector 50 executes a process of detecting various items of information relative to the pictures in accordance with the GOP header, the picture headers, and a plurality of flags of the picture coding extensions, to load the picture data into the ring buffer 5. However, this processing routine is extremely complex making it difficult to construct the stream detector 50.
The manner in which the stream detector 50 detects the completion of the loading process is explained in conjunction with the flow chart shown in FIG. 22. It is assumed in this flow chart that a random access is made to an entry sector written immediately anterior to an I-picture so that a proper picture can be obtained instantly in response to the random access.
At step S10, the stream detector searches for the picture.sub.-- start.sub.-- code in the picture.sub.-- header to detect the picture header of the I-picture and inquiry is made at step S12 to determine whether the picture.sub.-- start.sub.-- code has been detected. If the inquiry at step S12 is answered in the affirmative, that is, if the picture.sub.-- start.sub.-- code is detected, the operation proceeds to step S14. However, if the inquiry at step S12 is answered in the negative, that is, if no picture.sub.-- start.sub.-- code is detected, the process at step S12 is repeated until the picture.sub.-- start.sub.-- code is detected.
At step S14, the temporal reference is read from the detected picture header and its numerical value is stored in a register as TR0.
At step S16, another search (SRCH) for the picture.sub.-- start.sub.-- code in the picture header is made to detect the next picture and inquiry is made at step S18 to determine whether the picture.sub.-- start.sub.-- code has been detected. If the inquiry at step S18 is answered in the affirmative, that is, if the picture.sub.-- start.sub.-- code is detected, the operation proceeds to step S20. However, if the inquiry at step S18 is answered in the negative, the process at step S18 is repeated until the picture-start-code is detected.
Inquiry is made at step S20 to determine whether a GOP header has been detected in the picture start code, thereby to determine if the detected picture data is part of a pair. If the inquiry at step S20 is answered in the negative, that is, if no GOP header is detected, the operation proceeds to step S22. However, if this inquiry is answered in the affirmative, that is, if the GOP header is detected, the operation advances to step S26 because the existence of a GOP header between the picture data eliminates the possibility that these picture units are paired.
When the temporal reference is read from the detected picture header, its numerical value is stored in a register as TR1 as represented by step S22, and the operation advances to the inquiry at step S24 to determine whether the numerical values of the TR stored respectively in the register TR0 and TR1 are equal. If the inquiry at step S24 is answered in the affirmative, that is, if there is a coincidence of the two numerical values, the operation returns to step S16 and the processes discussed above in conjunction with steps S16 to S24 are repeated. It will be appreciated that the coincidence of two numerical values signifies that a picture data pair has been detected.
However, if the inquiry at step S24 is answered in the negative, that is, if the numerical values of the TR are not equal, the operation proceeds to step S26. Here, the picture header of the next picture is detected; and the picture coding type read from the picture header is stored in the register. The operation then advances to the inquiry at step S28 to determine whether the stored picture coding type represents a B-picture. If the inquiry at step S28 is answered in the affirmative, that is, if the detected picture is a B-picture, the operation returns to step S16 because the B-picture is not being sought; and the processes discussed above in conjunction with steps S16 to S28 are repeated to detect the next picture.
However, if the inquiry at step S28 is answered in the negative, that is, if the detected picture is not a B-picture, the temporal reference in the detected picture header is read and its numerical value is stored in the register as TR2, as represented by step S30. It will be appreciated that this detected picture is the first P-picture appearing after the I-picture.
At step S32, another search (SRCH) for the picture.sub.-- start.sub.-- code in the picture header is made to detect the next picture and inquiry is made at step S34 to determine whether the picture.sub.-- start.sub.-- code has been detected. If the inquiry at step S34 is answered in the affirmative, the operation proceeds to step S36. However, if inquiry at step S34 is answered in the negative, that is, if no picture.sub.-- start.sub.-- code has been detected, the process at step S34 is repeated until the picture.sub.-- start.sub.-- code is detected.
Inquiry is made at step S36 to determine whether a GOP header has been detected during the search for the picture.sub.-- start.sub.-- code, thereby to determine if the detected picture data is part of a pair. If the inquiry at step S36 is answered in the negative, that is, if no GOP header is detected, the operation proceeds to step S38. However, if the inquiry at step S36 is answered in the affirmative, that is, if a GOP header is detected, the operation advances to step S42 because the existence of a GOP header between picture units eliminates the possibility that these picture units are paired.
When the temporal reference is read from the detected picture header at step S38, its numerical value is stored in the register as TR3 and the operation advances to the inquiry at step S40 to determine whether a coincidence is attained between the numerical values of the TR stored respectively in the register as TR2 and TR3. If the inquiry at step S40 is answered in the affirmative, that is, if the two numerical values are equal, the operation returns to step S32 and the processes discussed above in conjunction with steps S32 to S40 are repeated. It will be appreciated that the coincidence of two numerical values signifies that a picture data pair has been detected.
However, if the inquiry at step S40 is answered in the negative, that is, if the numerical values of the TR are not coincident, the operation proceeds to step S42 to read the picture type. The operation then advances to the inquiry at step S44 to determine whether the stored picture coding type represents a B-picture. If the inquiry at step S44 is answered in the affirmative, that is, if the detected picture is a B-picture, the operation returns to step S32 because the B-picture is not being sought; and the processes discussed above in conjunction with steps S32 to S44 are repeated to detect the next picture.
However, if the inquiry at step S44 is answered in the negative, that is, if the detected picture is not a B-picture, the temporal reference detected in the picture header is read and its numerical value is stored in the register as TR4, as represented by step S46. It will be appreciated that this detected picture is the second P-picture appearing after the I-picture.
Proceeding to step S48, another search (SRCH) for the picture.sub.-- start.sub.-- code in the picture header is made to detect the next picture and inquiry is made at S50 to determine whether the picture.sub.-- start.sub.-- code has been detected. If the inquiry at step S50 is answered in the affirmative, that is, if the picture.sub.-- start.sub.-- code is detected, the operation proceeds to step S52. However, if the inquiry at step S50 is answered in the negative, the process at step S50 is repeated until the picture.sub.-- start.sub.-- code is detected.
Inquiry is made at step S52 to determine whether a GOP header has been detected during a search for the picture.sub.-- start.sub.-- code, thereby to determine if the detected picture data is part of a pair. If the inquiry at step S52 is answered in the negative, that is, if no GOP header is detected, the operation proceeds to step S54. However, if the inquiry at step S52 is answered in the affirmative, that is, if a GOP header is detected, the loading of the picture data into the ring buffer is completed and the process is terminated.
When the temporal reference is read from the detected picture header and its numerical value is stored in the register as TR5, as represented by step S54, the operation advances to the inquiry at step S56 to determine whether a coincidence is attained between the numerical values of the TR stored respectively as TR4 and TR5. If the inquiry at step S56 is answered in the affirmative, that is, if there is coincidence in the two numerical values, the operation returns to step S48 and the processes discussed above in conjunction with steps S48 to S56 are repeated. However, if the inquiry at step S56 is answered in the negative, that is, if the two numerical values of the TR are not equal, the loading of the picture data is complete and the process is terminated.
Thus, the stream detector 50 can load a bit stream by executing the above processing routine to load an I-picture and two succeeding P-pictures. However, as indicated by its lengthy description, it is quite onerous to execute this complex processing routine.